This invention relates to the capture of color motion imagery by using sequential frame samples of different color filtration, referred to as field sequential color capture.
Color lag is the result of capturing color components of an image at different times during exposure of a single frame. One way to solve this problem is to use multiple sensors, each capturing a color component. As will be explained, this approach has its own set of problems. There are, in the field of visual system modelling, three basic visual properties relating the luminance channel, as represented by video Y, and the opponent color channels, as approximated by the color difference signals, U and V. where Bxe2x88x92Y=U and Rxe2x88x92Y=V. These are:
1. The maximum temporal frequency response of the opponent color system is less than xc2xd that of the luminance.
2. The maximum spatial frequency response of the opponent color system is near xc2xd that of the luminance.
3. The maximum sensitivity of opponent color system is slightly greater than xc2xd that of the luminance.
The first and second properties are best described in A. B. Watson, Perceptual-components architecture for digital video, JOSA A V. 7 # 10, pp. 1943-1954, 1990; while the third property is described in K. T. Mullen. The contrast sensitivity of human colour vision to red-green and blue-yellow chromatic gratings J. Physiol. V. 359, pp. 381-400, 1985. These three properties will be used in the selection of the relative durations of exposures in order to prevent color lag and in the spatio-temporal integration following image capture.
L. Arend et al., Color breakup in sequentially scanned LCDs. SID 94 Digest, pp 201-204, 1994; and D. Post et al. Predicting color breakup on field-sequential displays, SPIE Proc. V. 3058, pp 57-65, 1997, specifically investigate color break-up, or color lag. Although the research behind these papers is intended for application to field sequential color displays, some of the findings are relevant to color field sequential capture (FSC). One of these is that color lag detection has a strong luminance component. In fact, as the color offset between R, G and B layers is increased from zero to detectability, the first signal to exceed threshold is the luminance. This manifests itself as blurring in texture before any high contrast edge blur occurs, because of the masking effects caused by high contrast edges. Eventually, as the offset is increased, color artifacts become visible at high contrast edges.
Hunt, The reproduction of colour in photography, printing, and television, 4th edition, pp. 409-410, Fountain Press, England, describes work done with YRB camera systems. These were primarily investigated for 3-sensor systems when the sensors where tubes. The YRB system attempted to reduce the visibility of the color offset problem due to manufacturing tolerances by optically aligning the tubes. Due to complications with gamma correction, it was abandoned in favor of a RGBY systems using four tubes. These complications, which were extremely difficult to resolve in analog systems, are now easily resolved in digital signal processing.
In order to appreciate the advantages of field sequential color capture, two other common approaches must be understood. One other approach is the use of three sensors, typically using red, green, and blue filtration, which simultaneously capture the scene""s dynamic content. This technique is used with both tube pickup devices and with 2D sensor arrays, such as charge coupled devices (CCD) or composite metal-on-silicon (CMOS) devices, which are referred to in the literature as 3-CCD cameras.
Another approach is the use of a single 2D sensor having color filtration applied separately to each pixel. The colors are arranged in spatially varying manners designed to provide a high spatial bandwidth for luminance or green, and to minimize color aliasing artifacts. The result is that each color layer has incomplete samples per frame, however, special interpolation algorithms are used to reconstruct full dimensioned frames for each color layer. This approach is known as color filter array (CFA) camera capture.
The 3-CCD, or 3-tube, approach""s chief disadvantage is the cost of three sensors. A second disadvantage is the problem of color mis-registration between the three sensors due to their alignment in the optical path relative to the scene, which may impose tight manufacturing tolerances that increase manufacturing costs. Color mis-registration may cause luminance blur in textures with very small amounts of mis-registration, and may cause color bleeding, also referred to as color fringing, at both achromatic and chromatic edges. If the registration is well aligned, the 3-CCD approach achieves the resolution of the sensors for all three color layers of a frame. Due to cost issues and manufacturing constraints, however, these approaches are only used for high-end studio video cameras, and digital still cameras aimed for the professional and advanced hobbyist.
The CFA approach is less costly because it uses only a single sensor, however, its disadvantages include reduced spatial resolution, the necessity for an interpolation algorithm to reconstruct the three color frames for display, and necessity for an anti-aliasing filter to prevent diagonal luminance high spatial frequencies from aliasing into lower frequency color patterns and to prevent color high spatial frequencies from aliasing into luminance or color patterns. Consequently, there is a trade-off between sharpness and color artifacts. These artifacts are quite noticeable in such common image content as highlight reflections in eyes, and the expected luminance high spatial frequencies in texture, e.g., hair, or geometric patterns. In current implementations, fairly complex interpolation algorithms, which include pattern recognition, are used to try to maximize sharpness and minimize color spatial artifacts. The sharpness/aliasing tradeoff may be described via FIG. 3, using either filter 42 or filter 44. Either filter may be increased in bandwidth by scaling their shape on the frequency axis. Though the image will appear to be sharper, signals having a frequency higher than that of the digital Nyquist will be aliased, i.e., the signal will fold over to frequencies lower than that of the digital Nyquist. These false lower frequencies are, in effect, added to the true signal. If all of the aliasing is removed, however, the image will appear blurred. It therefore requires a certain amount of craftsmanship in designing the filters to provide an appropriate amount of aliasing in order to maintain the bandwidths as high as possible. Most camera manufacturers opt to avoid any chromatic aliasing, because it is a new categorical distortion, and favor the sharpness reduction, which is already present to some extent. In summary, the two common approaches do not achieve the resolution of their sensors dimensions, either in luminance or in color.
The foregoing characteristics are depicted in FIG. 1-6. In FIG. 1, the Nyquist folding frequencies for the typical 2D CFA, also known as a Bayer pattern, are depicted, which are collectively referred to as the Nyquist boundary. Line 22, with a slope of xe2x88x921, is the boundary for green. The green boundary may achieve the value of 0.5 cycles/pixel (cy/pix) only at the horizontal and vertical frequencies. The red and blue components are shown by line 24 and are restricted to the square region limited by 0.25 cy/pix. Generally these color signals are manipulated so that G captures luminance, and R and B capture chrominance difference signals, by taking their difference or ratio with G. This means that luminance information with higher frequencies than those indicated in triangles 26, 28, which will alias into lower frequencies, showing up as both luminance and chrominance alias patterns. Similar effects occur for chrominance information outside of the smaller Nyquist square 24 boundary. Because it is most critical to get minimally aliased luminance information in an image, only the luminance Nyquist boundaries in the figures are discussed herein.
FIG. 2 includes several additional elements. FIG. 2a is a similar representation depicted in 3D. Contours 30 (FIG. 2 only) depict the human visual system contrast sensitivity function (CSF), which is analogous to its frequency response. This CSF is mapped to the digital frequency plane, based on calibration via the size of the image in pixel heights, and the viewing distance as expressed in units of picture heights. This mapping is a way of normalizing the actual physical display size and viewing distance. In FIG. 2, the displayed image size is 480 pixels high and the viewing distance is 6 picture heights (6H), which is the traditional NTSC image size. The CSF is a 2D surface, whose values are shown as a contour plot, with a peak frequency response near 0.08 cy/pix, for this viewing distance, which is six picture heights (H). The actual distance is dependent on the size of the display, however, xe2x80x9cnormalizationxe2x80x9d eliminates the need to include the actual viewing distance. The highest frequency contour 31 corresponds to the highest frequency the human eye can see.
The Nyquist boundary 32 for a full frame image capture system is depicted, which is the case for either 3-CCD systems or FSC color systems. This region is bounded by 0.5 cy/pix in both the horizontal and vertical directions, also referred to as xe2x80x9cshutter RGBxe2x80x9d. Area 34, located between the triangle formed by the luminance CFA boundary 22 and rectangle 32, is the resolution advantage of the full-frame over the CFA approaches. While this looks impressive, resulting in a doubling of the area of the CFA boundary, it is tempered by the visual system frequency response, whose anisotropy causes its isocontours to more closely follow the CFA pattern.
The final addition to FIG. 2 is the optical antialiasing filter which is used to prevent frequencies higher than the CFA Nyquist limit from reaching the sensor. If we assume the system""s focusing lens is used for invoking this function, it is generally isotropic, and to prevent aliasing at 45 degrees, we must select its bandwidth as shown by circular line 36. Note that some high vertical and horizontal frequencies that are actually lower than Nyquist limits imposed by the CFA are lost, as represented by area 38, located between CFA boundary 22 and optical cutoff filter boundary 26.
Lens optics are not generally used for antialiasing. A birefringent crystal filter is used because it has a better frequency response for a given cut-off frequency, as shown in FIG. 3, generally at 40. The optical transfer function (OTF) is the frequency response of the lens in use. In FIG. 3, the frequency response of an optical lens may at best be diffraction limited, and that response is shown as curve 42, which is designed to cut-off at the Nyquist of 0.5 cy/pix. The birefringent filter merely causes a displacement in the beam, resulting in two impulse responses in the spatial domain, which may be modeled as a cosine in the frequency domain. This response is shown as line 44, which is designed for the same cutoff frequency of 0.5 cy/pix.
Although the birefringent crystal filter has better frequency response characteristics for a given cut-off frequency, the usual technique is to have one crystal displace a pixel horizontally, and to then cascade another crystal which causes a vertical displacement, as may be seen in FIG. 3. The net effect is a Cartesian-separable low-pass filter (LPF) which is indicated in FIG. 4, generally at 50, where the cut-off frequencies are selected to avoid any luminance aliasing at the critical point at 45 degrees. Consequently, the horizontal and vertical Nyquist boundaries 52 end up at only 0.25 cy/pix, which strongly reduce the resolution potential of the sensor, thus reducing sharpness. Since various design aims are used to trade-off aliasing vs. sharpness, the typical CFA camera system frequency response will lie between square 52 and line 22, in two triangular regions 54. One may see the substantial increase in resolution area afforded by a full sensor frame capture system depicted by rectangle 56 intersecting at 0.5 cy/pix in the horizontal and vertical.
Much of the work in the FSC field was done prior to the NTSC color standard, when color field sequential was a competitor for color television. As a consequence, most of the prior art references are old. Analog field sequential color video was difficult to achieve at high frame rates and found its key applications in telecine and other specialized applications. Only recently has the activity picked up due to full digitization of the video system, which makes it easier to couple field sequential color capture to simultaneous color displays.
U.S. Pat. No. 2,738,377, to Weighton, granted Mar. 13, 1956 for Color Television, describes a color television system which uses a rotating color filter wheel, with equally spaced wedges in the order GRGBGRGB, with a single pickup tube, and a CRT to constitute a full color television system. The reference describes an eight-fold interlace, with the color wheel spinning fast enough to cause a different color filter for each interlace line. A sufficient number of lines are used to result in a captured image with 400 G lines and 200 R and B lines. The main difficulty with this approach is the extreme demand imposed on the system bandwidth due to the eightfold interlace. Another is that color field sequential display is required in this system, and because of eye movements, such displays are more susceptible to color breakup than color field sequential capture.
U.S. Pat. No. 3,604,839 to Kitsopoulos, granted Sep. 14, 1971 for Field-sequential color television apparatus employing color filter wheel and two camera tubes, describes a television system having a color filter wheel and two camera tubes, which is primarily aimed at telecine. The purpose of the two tubes is to allow for simultaneous capture of two different colors, thus allowing the exposure to lengthen, given the field rate constraints, and increase the signal to noise ratio.
U.S. Pat. No. 3,969,763 to Tan, granted Jul. 13, 1976, for Color television camera provided with a pickup device and color filter placed in front thereof, describes a color field sequential camera that uses a single pickup tube and a color filter wheel with many fine color strips. The purpose is to capture the color filtered sections of an image more spatially and temporally coincident. It approaches a color sequential interlace, and is primarily an analog hardware system addressing various delay and converter steps. The reference is notable because it discloses a system which uses liquid filters that are electronically controllable, rather than using a mechanical wheel. The system also captures in a YRB space, rather than the more common RGB.
U.S. Pat. No. 4,067,043, to Perry, granted Jan. 3, 1978 for Optical conversion method, describes the use of electro-optically controlling color filtration, in a field sequential mode, via the use of crossed polarizers.
U.S. Pat. No. 4,851,899 to Yoshida et al., granted Jul. 25, 1989, for Field-sequential color television camera, describes an RGB field sequential camera using a color wheel that causes the field order to be R1R2G1G2B1B2, etc. Because the first field of each color pair has a residual charge from the previous, different color, the system discards the first captured field in order to prevent color desaturation.
U.S. Pat. No. 4,967,264 to Parulski et al., granted Oct. 30, 1990, for Color sequential optical offset image sampling system, combines color field sequential concepts with sensor dithering to increase resolution. A color filter wheel is used containing, in order, 2 green, one red, and one blue element. The two green elements are placed at various tilts to create spatial offsets on the sensor, so the two green fields may be combined to create an image with higher green resolution than would be available solely from the sensor.
U.S. Pat. No. 5,084,761 to Nitta, granted Jan. 28, 1992, for Video camera with rotatable color filter and registration correction, addresses the color mis-registration caused by the variations in tilt and thickness of the color filter wheel in color field sequential cameras. The horizontal and vertical deflection signals are adjusted. Even though the color field sequential camera is, by design, not supposed to have a color registration problem, it does in fact have this problem because of the mechanics of the color wheel. The reference seeks to avoid this problem with electronically controlled color filters, such as a LCD.
U.S. Pat. No. 5,548,333 to Shibazaki et al., granted Aug. 20, 1996, for Color mixing prevention and color balance setting device and method for a field-sequential color television camera, addresses color mixing and color balance processes in a RGB filter wheel field sequential color camera. It allows a video field with a mixed color input, that is a transition from one color sector to another of the color wheel, to be used, but discards the signal during the mixed color-time interval.
Another goal of the system is to adjust-timing intervals to achieve white balance, also referred to as color automatic gain control (AGC). The timing intervals are determined from calibration with a white card to compensate for studio lighting color temperature. Due to the nature of the color wheel, the RGB exposure durations retain the same color lag, even if the exposure durations are shortened. This is illustrated up in FIG. 7, where the issues of discarded fields and color mixing have been simplified to highlight the exposure control, which is used for color temperature matching. Top row 56 depicts the exposure durations at a maximum per frame, with the horizontal axis being time, for a total of three frames. Bottom row 58 depicts a reduction in temporal exposure, where white bars indicate periods of no exposure, or non-capturing time, to the sensor accumulator. The start time of each color field is fixed in time due to the fixed patterns on the physical color wheel, so even though the durations are shortened, the color lag, as measured from either center-to-center positions, or from like edge positions, remains the same. A gap 59 is depicted as the non-capturing time between color components. Such a gap will be understood to also be present between each frame.
U.S. Pat. No. 5,748,236, to Shibazaki, granted May 5, 1998, for Color mixing prevention and color balance setting device and method for a field-sequential color television camera, is based on a continuation-in-part application from the ""333 patent, supra, however, this reference concentrates on the hardware, and specifically, the concept of discarding the accumulated charge during the mixed interval.
U.S. Pat. No. 5,751,384 to Sharp, granted May 12, 1998, for Color polarizers for polarizing an additive color spectrum along a first axis and its compliment along a second axis, describes switchable color filter via polarization and LCD as would be used in digital still and video cameras.
U.S. Pat. No. 5,767,899 to Hieda et al., granted Jun. 16, 1998, for Image pickup device, addresses the problem of constant luminance, where the luminance signal is formed prior to any gamma correction. This is opposed to conventional video processing, where the gamma correction nonlinearity is applied to the RGB values prior to the formation of the luminance signal. In this device the captured Y, Cr, and Cb outputs from a CFA interpolation process are converted to RGB, gamma corrected, then converted to conventional YR-YB-Y. Various base-clipping, or coring, limiting, and color suppression based on the luminance signal are then incorporated.
A method of field sequential color image capture includes optically capturing a scene; filtering the scene through an active color filter to produce first color components, thereby modifying the spectral transmission of the scene as a function of time; detecting the scene with a single, monochrome sensor and dividing the scene into multiple second color components, whereby each first color component of the scene is detected at a different point in time; aligning the second color components in time for each frame interval; storing each color component in a memory unit; combining the stored second color components into a frame image; and processing the frame image for color reproduction and format.
A system for capturing a field sequential color image, includes an optical capture mechanism for capturing a scene frame-by-frame in frame intervals; an active color filter for producing multiple first color components; a single monochrome area sensor; an array field selector for dividing the scene into multiple second color components; multiple memory locations for storing each color component; a field-to-frame combiner for combining the stored second color components into a frame image; and a color reproduction processor for processing the frame image for color reproduction and format.
An object of this invention is to provide a color image capture system and method using only a single sensor while capturing the full resolution of the sensor for all three color planes.
Another object is to provide a system and method wherein color lag will decrease as shutter durations decrease.
A further object of the invention is to provide a system and method that reduce color mis-registration.